Foreign object identification and image augmentation for intraoral scanning

ABSTRACT

In a method of generating a virtual 3D model of a dental site, scan data comprising a plurality of images of a dental site is received during an intraoral scan. An analysis of an image is performed. A representation of a reference object with known properties is identified in the image based on the analysis. A virtual 3D model of the dental site is generated based on the plurality of images. The image and/or the virtual 3D model of the dental site is modified by adding additional data about the reference object to the intraoral image and/or the virtual 3D model based on the one or more known properties of the reference object. If the image is modified, it may be modified prior to generation of the virtual 3D model, and the virtual 3D model may be generated using the modified image.

RELATED APPLICATIONS

This patent application claims the benefit under 35 U.S.C. § 119(e) ofU.S. Provisional Application No. 62/815,954, filed Mar. 8, 2019, whichis herein incorporated by reference.

TECHNICAL FIELD

Embodiments of the present disclosure relate to the field of intraoralscanning and, in particular, to a system and method for performingreal-time filtering of intraoral images generated during intraoralscanning.

BACKGROUND

In prosthodontic procedures designed to insert a dental prosthesis inthe oral cavity, the dental site at which the prosthesis is to beimplanted in many cases should be measured accurately and studiedcarefully, so that a prosthesis such as a crown, denture or bridge, forexample, can be properly designed and dimensioned to fit in place. Agood fit enables mechanical stresses to be properly transmitted betweenthe prosthesis and the jaw, and to prevent infection of the gums via theinterface between the prosthesis and the dental site, for example.

Some procedures also call for removable prosthetics to be fabricated toreplace one or more missing teeth, such as a partial or full denture, inwhich case the surface contours of the areas where the teeth are missingneed to be reproduced accurately so that the resulting prosthetic fitsover the edentulous region with even pressure on the soft tissues.

In some practices, the dental site is prepared by a dental practitioner,and a positive physical model of the dental site is constructed usingknown methods. Alternatively, the dental site may be scanned to provide3D data of the dental site (i.e. in the form of intraoral images such asheight maps). In either case, the virtual or real model of the dentalsite is sent to the dental lab, which manufactures the prosthesis basedon the model. However, if the model is deficient or undefined in certainareas, or if the preparation was not optimally configured for receivingthe prosthesis, the design of the prosthesis may be less than optimal.For example, if the insertion path implied by the preparation for aclosely-fitting coping would result in the prosthesis colliding withadjacent teeth, the coping geometry has to be altered to avoid thecollision, which may result in the coping design being less optimal.Further, if the area of the preparation containing a finish line lacksdefinition, it may not be possible to properly determine the finish lineand thus the lower edge of the coping may not be properly designed.Indeed, in some circumstances, the model is rejected and the dentalpractitioner then re-scans the dental site, or reworks the preparation,so that a suitable prosthesis may be produced.

In orthodontic procedures it can be important to provide a model of oneor both jaws. Where such orthodontic procedures are designed virtually,a virtual model of the oral cavity is also beneficial. Such a virtualmodel may be obtained by scanning the oral cavity directly, or byproducing a physical model of the dentition, and then scanning the modelwith a suitable scanner.

Thus, in both prosthodontic and orthodontic procedures, obtaining athree-dimensional (3D) model of a dental site in the oral cavity is aninitial procedure that is performed. When the 3D model is a virtualmodel, the more complete and accurate the scans of the dental site are,the higher the quality of the virtual model, and thus the greater theability to design an optimal prosthesis or orthodontic treatmentappliance(s).

During an intraoral scanning session, often there are foreign objectsthat are disposed within a patient's mouth. Such foreign objects mayreduce an accuracy of the generated virtual model. For example, imageprocessing algorithms may adjust the shape of one or more regions of adental arch based on the shape of the foreign object under theassumption that the foreign object is a part of the dental arch.Additionally, foreign objects may move between the generation ofdifferent intraoral images, which may interfere with image registration.

SUMMARY

In a first aspect of the disclosure, a method comprises receiving scandata comprising an intraoral image during an intraoral scan of a dentalsite, identifying a representation of a foreign object in the intraoralimage based on an analysis of the scan data, modifying the intraoralimage by removing the representation of the foreign object from theintraoral image, receiving additional scan data comprising a pluralityof additional intraoral images of the dental site during the intraoralscan, and generating a virtual three-dimensional (3D) model of thedental site using the modified intraoral image and the plurality ofadditional intraoral images.

A second aspect of the disclosure may extend the first aspect of thedisclosure. In the second aspect of the disclosure, the method furtherincludes identifying an additional representation of the foreign objectin one of more additional intraoral images of the plurality ofadditional intraoral images of the dental site, and modifying the one ormore additional intraoral images by removing the additionalrepresentation of the foreign object from the one or more additionalintraoral images, wherein the foreign object is a stationary object thathas a same position at the dental site in the intraoral image and theone or more additional intraoral images.

A third aspect of the disclosure may extend the first or second aspectof the disclosure. In the third aspect of the disclosure, the methodfurther includes analyzing the scan data to determine at least one of areflectivity of the foreign object, a diffraction of the foreign object,a reactivity of the foreign object to a wavelength of light, a textureof the foreign object, a surface pattern of the foreign object, a colorof the foreign object or a shape of the foreign object; and identifyingthe representation of the foreign object based on at least one of thereflectivity of the foreign object, the diffraction of the foreignobject, the reactivity of the foreign object to a wavelength of light,the texture of the foreign object, the surface pattern of the foreignobject, the color of the foreign object or the shape of the foreignobject.

A fourth aspect of the disclosure may extend the first through thirdaspects of the disclosure. In the fourth aspect of the disclosure, themethod further includes performing image processing on the intraoralimage to determine a plurality of shapes in the intraoral image, theplurality of shapes comprising the shape of the foreign object,comparing the plurality of shapes in the intraoral image to a pluralityof known shapes of reference objects, and determining that the shape ofthe foreign object matches a known shape of a reference object as aresult of the comparing.

A fifth aspect of the disclosure may extend the fourth aspect of thedisclosure. In the fifth aspect of the disclosure, the method furtherincludes performing the following before the intraoral scan of thedental site: entering a training mode, receiving a plurality of imagesof the foreign object during the training mode, generating a virtualmodel of the foreign object based on the plurality of images, and addingat least one of the virtual model of the foreign object or the pluralityof images of the foreign object to a reference object library thatcomprises the plurality of known shapes of the reference objects.

A sixth aspect of the disclosure may extend the first through fifthaspect of the disclosures. In the sixth aspect of the disclosure, themethod further includes performing image processing of the intraoralimage to identify at least one of a plurality of textures or a pluralityof surface patterns in the intraoral image, determining at least one ofa texture of the plurality of textures or a surface pattern of theplurality of surface patterns that is not naturally occurring at thedental site, and determining a contour of a region of the intraoralimage having at least one of the texture or the surface pattern that isnot naturally occurring at the dental site, wherein the region of theintraoral image within the contour is removed from the intraoral image.

A seventh aspect of the disclosure may extend the sixth aspect of thedisclosure. In the seventh aspect of the disclosure, determining atleast one of the texture or the surface pattern that is not naturallyoccurring at the dental site comprises at least one of: a) comparing theplurality of textures identified in the intraoral image to a pluralityof known textures of reference objects, and determining that the atleast one texture matches a known texture of a reference object as aresult of the comparing; b) comparing the plurality of surface patternsidentified in the intraoral image to a plurality of known surfacepatterns of reference objects, and determining that the at least onesurface patterns matches a known surface pattern of a reference objectas a result of the comparing; c) comparing the plurality of texturesidentified in the intraoral image to a plurality of known textures thatnaturally occur at the dental site, and determining that the at leastone texture fails to match any known texture that naturally occurs inthe mouth as a result of the comparing; or d) comparing the plurality ofsurface patterns identified in the intraoral image to a plurality ofknown surface patterns that naturally occur at the dental site, anddetermining that the at least one surface pattern fails to match anyknown surface pattern that naturally occurs in the mouth as a result ofthe comparing.

An eighth aspect of the disclosure may extend the first through seventhaspects of the disclosure. In the eighth aspect of the disclosure, themethod further includes performing image processing of the intraoralimage to identify a plurality of colors in the intraoral image,determining at least one color of the plurality of colors that is notnaturally occurring at the dental site, and determining a contour of aregion of the intraoral image having the at least one color that is notnaturally occurring at the dental site, wherein the region of theintraoral image within the contour is removed from the intraoral image.

A ninth aspect of the disclosure may extend the eighth aspect of thedisclosure. In the ninth aspect of the disclosure, the method furtherincludes determining the at least one color that is not naturallyoccurring at the dental site comprises performing the following for eachpixel or voxel of the intraoral image, determining, for each colorchannel of a color encoding system, at least one of a saturation valueor an intensity value, determining a tuple comprising at least one ofthe saturation value or the intensity value for each color channel ofthe color encoding system, determining that the tuple is outside of aspecified color range, and determining a contour of the region in theintraoral image having the at least one color that is not naturallyoccurring comprises determining a plurality of contiguous pixels orvoxels having tuples that are outside of the specified color range.

A tenth aspect of the disclosure may extend the first through ninthaspects of the disclosure. In the tenth aspect of the disclosure, themethod further includes receiving an indication that the foreign objectis at the dental site, the indication comprising an identification ofthe foreign object, querying a reference object library using theidentification of the foreign object; receiving a response to the query,the response comprising one or more known properties of the foreignobject, the one or more known properties comprising at least one of aknown reflectivity of the foreign object, a known diffraction of theforeign object, a known reactivity of the foreign object to thewavelength of light, a known texture of the foreign object, a knownsurface pattern of the foreign object, a known color of the foreignobject or a known shape of the foreign object; and using the one or moreknown properties of the foreign object to identify the representation ofthe foreign object in the intraoral image.

An eleventh aspect of the disclosure may extend the first through tenthaspects of the disclosure. In the eleventh aspect of the disclosure, themethod further includes, during the intraoral scan, generating a view ofthe intraoral site based on the modified intraoral image and the one ormore additional intraoral images; generating an outline of the foreignobject; adding the outline of the foreign object to the view; andproviding a user interface that enables a user to select the outline ofthe foreign object, wherein user selection of the outline of the foreignobject causes the representation of the foreign object to be added backto the intraoral image and further causes the foreign object to be addedto a reference object library of known objects not to be filtered out.

A twelfth aspect of the disclosure may extend the first through eleventhaspects of the disclosure. In the twelfth aspect of the disclosure, themethod further includes performing the following prior to modifying theintraoral image: determining a confidence value associated with therepresentation of the foreign object; determining that the confidencevalue is below a confidence threshold; presenting an option to a) removethe representation of the foreign object from the intraoral image or b)leave the representation of the foreign object in the intraoral image;and receiving a user selection to remove the representation of theforeign object from the intraoral image.

A thirteenth aspect of the disclosure may extend the twelfth aspect ofthe disclosure. In the thirteenth aspect of the disclosure, the methodfurther includes determining one or more properties of the foreignobject, the one or more properties comprising at least one of areflectivity of the foreign object, a diffraction of the foreign object,a reactivity of the foreign object to the wavelength of light, a textureof the foreign object, a surface pattern of the foreign object, a colorof the foreign object or a shape of the foreign object; and adding anentry for the foreign object to a reference object library of knownforeign objects to be filtered out.

A fourteenth aspect of the disclosure may extend the twelfth orthirteenth aspect of the disclosure. In the fourteenth aspect of thedisclosure, identifying the representation of the foreign objectcomprises: processing the intraoral image using a machine learning modelthat has been trained to identify foreign objects at dental sites; andreceiving an output of the machine learning model, wherein the outputcomprises a binary mask that has a number of entries that is equal to anumber of pixels or voxels in the intraoral image, wherein entriesassociated with pixels or voxels that are part of the foreign objecthave a first value and wherein entries associated with pixels or voxelsthat are not part of the foreign object have a second value; wherein theintraoral image is modified using the binary mask.

In a fifteenth aspect of the disclosure, a method comprises receivingintraoral scan data comprising a plurality of images of a dental site;generating a virtual three-dimensional (3D) model of the dental sitebased on the plurality of images; performing an analysis on an image ofthe dental site; identifying a representation of a reference object inthe image, wherein the reference object has one or more knownproperties; and modifying at least one of the image or the virtual 3Dmodel of the dental site by adding additional data about the referenceobject to at least one of the image or the virtual 3D model based on theone or more known properties of the reference object.

A sixteenth aspect of the disclosure may extend the fifteenth aspect ofthe disclosure. In the sixteenth aspect of the disclosure, the image isone of the plurality of images of the dental site used to generate thevirtual 3D model, the image is modified by adding the additional dataabout the reference object to the image, and the image, as modified, isused to generate the virtual 3D model of the dental site.

A seventeenth aspect of the disclosure may extend the sixteenth aspectof the disclosure. In the seventeenth aspect of the disclosure, themethod further comprises performing image processing on the image todetermine a probable object represented in the image, wherein theprobable object comprises one or more physical properties; comparing theone or more physical properties of the probable object to the one ormore known properties of the reference object; determining that the oneor more physical properties of the probable object match the one or moreknown properties the reference object as a result of the comparing; anddetermining that the probable object is the reference object.

An eighteenth aspect of the disclosure may extend the seventeenth aspectof the disclosure. In the eighteenth aspect of the disclosure, the oneor more physical properties comprise at least one of a reflectivity, adiffraction, a reactivity to a wavelength of light, a texture, a surfacepattern, a color or a shape.

A nineteenth aspect of the disclosure may extend any of the fifteenththrough the eighteenth aspect of the disclosure. In the nineteenthaspect of the disclosure, performing the analysis of the image comprisesinputting the image into a machine learning model trained to identifyone or more types of reference objects, wherein the machine learningmodel outputs an indication of the representation of the referenceobject in the image.

A twentieth aspect of the disclosure may extend the fifteenth ornineteenth aspect of the disclosure. In the twentieth aspect of thedisclosure, the method further comprises generating the image on whichthe analysis is performed by projecting the virtual 3D model onto aplane.

A twenty first aspect of the disclosure may extend the twentieth aspectof the disclosure. In the twenty first aspect of the disclosure, themethod further comprises: during an intraoral scan, generating a view ofthe dental site based on the modified image, wherein the view comprisesan outline of the reference object based on the additional data aboutthe reference object; receiving a plurality of additional images of thedental site during the intraoral scan; replacing one or more parts of ashape of the reference object based on the plurality of additionalimages; and updating the view of the dental site, the updated viewshowing those parts of the shape of the reference object that have beenreplaced.

A twenty second aspect of the disclosure may extend any of the fifteenththrough the twenty first aspect of the disclosure. In the twenty secondaspect of the disclosure, identifying the representation of thereference object comprises processing the image using a machine learningmodel that has been trained to identify one or more foreign objects atdental sites; and receiving an output of the machine learning model,wherein the output comprises a binary mask that has a number of entriesthat is equal to a number of pixels or voxels in the image, whereinentries associated with pixels or voxels that are part of the referenceobject have a first value and wherein entries associated with pixels orvoxels that are not part of the reference object have a second value.

A twenty third aspect of the disclosure may extend any of the fifteenththrough the twenty second aspect of the disclosure. In the twenty thirdaspect of the disclosure, the method further comprises performing thefollowing before receiving the intraoral scan data: entering a trainingmode; receiving a plurality of images of the reference object during thetraining mode; generating a virtual model of the reference object basedon the plurality of images of the reference object; and adding at leastone of the virtual model of the reference object or the plurality ofimages of the reference object to a reference object library thatcomprises entries for a plurality of reference objects.

A twenty fourth aspect of the disclosure may extend any of the fifteenththrough the twenty third aspect of the disclosure. In the twenty fourthaspect of the disclosure, the method further comprises receiving anindication that the reference object is at the dental site, theindication comprising an identification of the reference object;querying a reference object library comprising entries for a pluralityof reference objects using the identification of the reference object;receiving a response to the query, the response comprising dataassociated with the reference object; and using the data to identify thereference object in the image.

A twenty fifth aspect of the disclosure may extend any of the fifteenththrough the twenty fourth aspect of the disclosure. In the twenty fifthaspect of the disclosure, the one or more known properties comprise atleast one of a known reflectivity of the reference object, a knowndiffraction of the reference object, a known reactivity of the referenceobject to a wavelength of light, a known texture of the referenceobject, a known surface pattern of the reference object, a known colorof the reference object or a known shape of the reference object.

A twenty sixth aspect of the disclosure may extend any of the fifteenththrough the twenty fifth aspect of the disclosure. In the twenty sixthaspect of the disclosure, the method further comprises determining aconfidence value associated with the reference object; determining thatthe confidence value is below a confidence threshold; presenting anoption to a) add the additional data about the reference object to atleast one of the image or the virtual 3D model or b) leave arepresentation of the reference object in at least one of the image orthe virtual 3D model unchanged; and receiving a user selection to addthe additional data about the reference object to at least one of theimage or the virtual 3D model.

A twenty seventh aspect of the disclosure may extend any of thefifteenth through the twenty sixth aspect of the disclosure. In thetwenty seventh aspect of the disclosure, the reference object comprisesat least one of a shape or a material that causes intraoral scans of thereference object to have a reduced accuracy.

A further aspect of the disclosure includes a computer readable mediumcomprising instructions that, when executed by a processing device,cause the processing device to perform the method of one or more of thefirst through twenty seventh aspects of the disclosure set forth above.

A further aspect of the disclosure includes a system comprising ahandheld scanner to perform an intraoral scan, and a non-transitorycomputer readable medium comprising instructions that, when executed bya processor, cause the process to perform operations of the method ofone or more of the first through twenty seventh aspects of thedisclosure.

A further aspect of the disclosure includes a system comprising ahandheld scanner to perform an intraoral scan, and a computing devicethat executes the method of one or more of the first through twentyseventh aspects of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1A illustrates one embodiment of a system for performing intraoralscanning and generating a virtual three dimensional model of a dentalsite.

FIG. 1B illustrates a flow diagram for a method of identifyingrepresentations of foreign objects in intraoral images, in accordancewith embodiments of the present disclosure.

FIG. 2 illustrates a flow diagram for a method of filteringrepresentations of foreign objects out of intraoral images, inaccordance with embodiments of the present disclosure.

FIG. 3 illustrates a flow diagram for a method of identifying foreignobjects in intraoral images, in accordance with embodiments of thepresent disclosure.

FIG. 4 illustrates a flow diagram for a method of adding an entry for aforeign object to a foreign object reference library, in accordance withembodiments of the present disclosure.

FIG. 5 illustrates a flow diagram for a method of using a foreign objectreference library to identify a foreign object in an intraoral image, inaccordance with embodiments of the present disclosure.

FIG. 6 illustrates a flow diagram for a method of identifying foreignobjects in intraoral images based on color, in accordance withembodiments of the present disclosure.

FIG. 7 illustrates a flow diagram for a method of providing a userinterface for management of foreign objects in intraoral images, inaccordance with embodiments of the present disclosure.

FIG. 8 illustrates a flow diagram for a further method of providing auser interface for management of foreign objects in intraoral images, inaccordance with embodiments of the present disclosure.

FIG. 9 illustrates a flow diagram for a method of filteringrepresentations of foreign objects out of intraoral images based on useof a trained machine learning model, in accordance with embodiments ofthe present disclosure.

FIG. 10 illustrates a portion of an example dental arch with a foreignobject disposed thereon.

FIG. 11 illustrates a first intraoral image that includes a firstrepresentation of the foreign object, in accordance with one embodimentof the present disclosure.

FIG. 12 illustrates a modified version of the first intraoral image ofFIG. 11, where the foreign object has been filtered out in the modifiedversion of the first intraoral image, in accordance with one embodimentof the present disclosure.

FIG. 13A illustrates view of a portion of an example dental arch with aforeign object disposed thereon, in accordance with one embodiment ofthe present disclosure.

FIG. 13B illustrates the view of FIG. 13A, where not yet scanned areasof the foreign object are down, in accordance with one embodiment of thepresent disclosure.

FIG. 14 illustrates a block diagram of an example computing device, inaccordance with embodiments of the present disclosure.

FIG. 15 illustrates a method of orthodontic treatment using a pluralityof appliances, in accordance with embodiments.

FIG. 16 illustrates a method for designing an orthodontic appliance, inaccordance with embodiments.

FIG. 17 illustrates a method for digitally planning an orthodontictreatment, in accordance with embodiments.

DETAILED DESCRIPTION

Described herein is a method and apparatus for improving the quality ofintraoral scans taken of dental sites that include one or more foreignobjects. There are often interfering situations during intraoralscanning, where each interfering situation includes the presence of oneor more foreign objects in intraoral images generated during anintraoral scan. The intraoral images may be or include height maps insome embodiments. For example, an intraoral image may be a 2D height mapthat includes a depth value for each pixel in the height map. An exampleof an intraoral scanner that generates such height maps during intraoralscanning is the iTero® intraoral digital scanner manufactured by AlignTechnology, Inc.

Foreign objects in intraoral scans (e.g., in intraoral images and/orheight maps) may be objects that naturally occur in the mouth but thatare not part of the dental arch being scanned (e.g., such as saliva,blood, tongue, bubbles, etc.). Such foreign objects may also be objectsthat do not naturally occur in the mouth (e.g., such as a cotton roll, afinger of an operator of an intraoral scanner, a suction device, an airblowing device, a dental mirror, a cord, a hand, a glove, and so on).The occurrence of foreign objects in the intraoral scan data (i.e.intraoral images) can slow down processing of those intraoral images andreduce the accuracy of a virtual 3D model generated from the intraoralimages. Accordingly, embodiments described herein provide techniques forfiltering out (i.e. removing) the depictions of foreign objects inreal-time or near real-time (e.g., as the images are generated) beforethose foreign objects are used for image registration and/or virtualmodel generation. By removing the foreign objects from the intraoralimages before performing further processing on those intraoral images,the speed and/or accuracy of the further processing may be increased.For example, the speed of image registration may be increased, theaccuracy of image registration may be increased, the number of instancesof failures to register images may be decreased, the accuracy of agenerated 3D virtual model may be increased, and the speed of generatingthe 3D virtual model may be increased. Additionally, process intensiveoperations of accounting for and mitigating problems caused by foreignobjects may be omitted, further increasing the speed of imageprocessing.

Notably, the detection of foreign objects and removal of such foreignobjects from intraoral images may be performed using a single image andmay be performed on static (i.e. non-moving) objects. This is incontrast to many traditional techniques for identification and removalof unwanted moving objects from images (referred to as space carvingtechniques). Such traditional space carving techniques that identify andremove moving objects cannot work on a single image and do not work forobjects that do not move between multiple frames of a video or multipleimages of an area. In contrast, the techniques described herein mayidentify and remove stationary objects and may work on a single image.

In other embodiments, a known reference object may be identified in animage of a dental site (e.g., an intraoral image generated by anintraoral scanner or an image generated by projecting a virtual 3D modelof a dental site onto a 3D surface or plane) and/or in a virtual 3Dmodel of the dental site. The image and/or virtual 3D model of thedental site may then be modified by adding additional information aboutthe reference object to the image and/or virtual 3D model based on knownproperties of the reference object.

In one embodiment, a computer-implemented method comprises receivingscan data comprising an intraoral image during an intraoral scan of adental site and identifying a representation of a foreign object in theintraoral image based on an analysis of the scan data. Thecomputer-implemented method further comprises modifying the intraoralimage by removing the representation of the foreign object from theintraoral image. The computer-implemented method further comprisesreceiving additional scan data comprising a plurality of additionalintraoral images of the dental site during the intraoral scan. Thecomputer-implemented method further comprises generating a virtualthree-dimensional (3D) model of the dental site using the modifiedintraoral image and the plurality of additional intraoral images.

In one embodiment, a non-transitory computer readable medium comprisesinstructions that, when executed by a processing device, cause theprocessing device to perform sequence of operations. The operationsinclude receiving an intraoral image during an intraoral scan of adental site and performing image processing on the intraoral image todetermine a plurality of probable objects in the intraoral image,wherein each of the plurality of probable objects comprises one or morephysical properties. The operations further include comparing, for eachof the plurality of probable objects, the one or more physicalproperties of the probable object to a plurality of known properties ofreference objects. The operations further include determining that oneor more physical properties of a probable object of the plurality ofprobable objects match one or more known properties a reference objectas a result of the comparing. The operations further include determiningthat the probable object is a foreign object at the intraoral site andthen modifying the intraoral image by adding additional data about theforeign object to the intraoral image based on the one or more knownproperties of the reference object.

In one embodiment, a method of generating a virtual 3D model of a dentalsite includes receiving intraoral scan data comprising a plurality ofimages of a dental site. The method further includes generating avirtual three-dimensional (3D) model of the dental site based on theplurality of images. The method further includes performing an analysison an image of the dental site. The image may be one of the imagesincluded in the received intraoral scan data. Alternatively, the imagemay be generated by projecting the virtual 3D model of the dental siteonto a 2D surface or plane. The image may be or include a height map ofthe dental site. The method further includes identifying arepresentation of a reference object in the image, wherein the referenceobject has one or more known properties. The method further includesmodifying at least one of the image or the virtual 3D model of thedental site by adding additional data about the reference object to atleast one of the image or the virtual 3D model based on the one or moreknown properties of the reference object. The enables images and virtual3D models to be improved by augmenting the image and/or virtual 3D modelwith accurate information about known reference objects. Thus, theaccuracy of reference objects that are difficult to scan (e.g., thosemade of titanium) may be improved in images and virtual 3D models usingtechniques described herein.

In one embodiment, a system includes a handheld scanner to perform anintraoral scan and a non-transitory computer readable medium comprisinginstructions that, when executed by a processor, cause the process toperform the above described sequence of operations.

FIG. 1A illustrates one embodiment of a system 100 for performingintraoral scanning and/or generating a virtual three-dimensional (3D)model of a dental site. In one embodiment, system 100 carries out one ormore operations of the below methods described with reference to FIGS.1B-9. System 100 includes a computing device 105 that may be coupled toa scanner 150 and/or a data store 110.

Computing device 105 may include a processing device, memory, secondarystorage, one or more input devices (e.g., such as a keyboard, mouse,tablet, and so on), one or more output devices (e.g., a display, aprinter, etc.), and/or other hardware components. Computing device 105may be connected to a data store 110 either directly or via a network.The network may be a local area network (LAN), a public wide areanetwork (WAN) (e.g., the Internet), a private WAN (e.g., an intranet),or a combination thereof. The computing device 105 and/or the data store110 may be integrated into the scanner 150 in some embodiments toimprove performance and mobility.

Data store 110 may be an internal data store, or an external data storethat is connected to computing device 105 directly or via a network.Examples of network data stores include a storage area network (SAN), anetwork attached storage (NAS), and a storage service provided by acloud computing service provider. Data store 110 may include a filesystem, a database, or other data storage arrangement.

In some embodiments, a scanner 150 for obtaining three-dimensional (3D)data of a dental site in a patient's oral cavity (referred to as anintraoral scanner) is also operatively connected to the computing device105. Scanner 150 may include a probe (e.g., a hand held probe) foroptically capturing three dimensional structures (e.g., by confocalfocusing of an array of light beams). One example of such a scanner 150is the iTero® intraoral digital scanner manufactured by AlignTechnology, Inc. Other examples of intraoral scanners include the 3M™True Definition Scanner and the Apollo DI intraoral scanner and CEREC ACintraoral scanner manufactured by Sirona®.

The scanner 150 may be used to perform an intraoral scan of a patient'soral cavity. A result of the intraoral scan may be a sequence ofintraoral images that have been discretely generated (e.g., by pressingon a “generate image” button of the scanner for each image).Alternatively, a result of the intraoral scan may be one or more videosof the patient's oral cavity. An operator may start recording the videowith the scanner 150 at a first position in the oral cavity, move thescanner 150 within the oral cavity to a second position while the videois being taken, and then stop recording the video. In some embodiments,recording may start automatically as the scanner identifies that it hasbeen inserted into a patient oral cavity. The scanner 150 may transmitthe discrete intraoral images (e.g., height maps) or intraoral video(referred to collectively as image data 135) to the computing device105. Note that in some embodiments the computing device may beintegrated into the scanner 150. Computing device 105 may store theimage data 135 in data store 110. Alternatively, scanner 150 may beconnected to another system that stores the image data in data store110. In such an embodiment, scanner 150 may not be connected tocomputing device 105.

Computing device 105 may include a foreign object identifier 115, animage registration module 120, and/or a model generation module 125. Themodules 115, 120, 125 may be software modules that are components of asingle software application (e.g., different libraries of a singleapplication), may be distinct software applications, may be firmware,may be hardware modules, and/or a combination thereof.

Foreign object identifier 115 may analyze received image data 135 (e.g.,discrete intraoral images or intraoral images that are frames of anintraoral video) using one or more of the techniques described hereinbelow to identify foreign objects in the intraoral images of the imagedata 135. In some embodiments, the data store 110 may include areference object library that includes entries for one or more referenceobjects 130. The reference objects 130 may include well known equipmentthat is standard in dentist offices and/or orthodontist offices. Suchreference objects 130 may be default reference objects that the datastore 110 is prepopulated with. Some reference objects 130 may be customreference objects that may have been generated by a dental practitionerusing a training mode. Each entry for a reference object may include oneor more physical properties of that reference object, such as a shape(e.g., a 3D virtual model of the reference object), color, texture,surface pattern, and so on. Some techniques for identifying foreignobjects used by foreign object identifier 115 include determiningphysical properties of objects in a received intraoral image andcomparing those physical properties of reference objects 130 in thereference object library. If a match is found, then an objectrepresented in the intraoral image may be identified as a particularforeign object associated with the match.

In one embodiment a user may cause the foreign object identifier 115 toenter a training mode. During the training mode, the user may use thescanner 150 to perform a 3D scan of an object. Images of the 3D objectmay then be used to generate a virtual 3D model of the foreign object.An entry for the scanned foreign object may then be added to thereference objects 130. The entry may be for the same object and/or forother similar objects. The entry may include the generated virtual 3Dmodel, a name of the object and/or one or more physical properties ofthe object.

If foreign object identifier 115 identifies a foreign object in anintraoral image, then foreign object identifier 115 may either filterout the foreign object from the intraoral image, leave the foreignobject unchanged, or add further detailed information about the foreignobject to the intraoral image (or otherwise augment the representationof the foreign object in the intraoral image).

Image registration module 120 registers the intraoral images that havebeen processed by the foreign object identifier 115 using one or moreimage registration techniques. For example, image registration module120 may perform image registration using modified intraoral images inwhich representations of foreign objects have been removed from theintraoral images. After image registration is complete, or as imageregistration is performed, model generation module 125 generates a 3Dvirtual model of the imaged dental site.

A user interface 140 may function during an intraoral scan session. Asscanner 150 generates intraoral images and those intraoral images areprocessed by foreign object identifier 115, a view screen showing atwo-dimensional (2D) or 3D representation of a scanned dental site maybe output to a display (e.g., of computing device 105). Foreign objectsthat have been filtered out by foreign object identifier 115 may not beshown in the 2D or 3D representation. Images may be stitched togetherduring the intraoral scan session to continuously update therepresentation of the dental site during the intraoral scan session.Image registration processes and model generation processes may beperformed similar to those performed by image registration module 120and model generation module 125, except more quickly and with loweraccuracy. The representation of the dental site may show a user of thescanner 150 what areas have been sufficiently imaged and which areaswould benefit from further imaging.

In some instances, foreign object identifier 115 may be unable toaccurately determine whether an object at a dental site is a foreignobject or a native object. In such instances, the foreign objectidentifier 115 may or may not filter out the foreign object. If theforeign object is filtered out, then the user interface may still showan outline, black and white representation, translucent representation,mesh representation, or other non-standard representation of the foreignobject in the view screen. A user may then select the representation ofthe foreign object shown in the view screen and be presented with anoption to proceed with filtering out the foreign object or to add theforeign object back to the intraoral images. If the user selects tofilter out the foreign object, then the outline or other non-standardrepresentation of the foreign object may be removed from the viewscreen. Additionally, data (e.g., physical properties) about the foreignobject may be added to an entry for a reference object 130. Future scansof the same patient and/or different patients may then use the newreference object 130 and the data associated with it to successfullyidentify additional instances of the foreign object. Therefore, theaccuracy of the foreign object identifier may continually improve. Ifthe user selects not to filter out the foreign object, then the foreignobject may be added back to the intraoral image (or the originalintraoral image may have been kept and may be used rather than themodified version of the intraoral image). If the user fails to selectthe foreign object, then the original decision to filter out the foreignobject may stand. In such instances, a new reference object may or maynot be added to data store 110 for the foreign object.

If the foreign object was not originally filtered out, then the foreignobject will be shown in the view screen in the same manner as otherfeatures and objects of the dental site. However, the foreign object maybe outlined, highlighted, or otherwise emphasized. The user may thenselect the foreign object representation in the view screen and bepresented with an option to filter out the foreign object or to leavethe foreign object in the intraoral images. If the user selects tofilter out the foreign object, then the foreign object may be removedfrom intraoral images of the intraoral scan that depict the foreignobject. Additionally, data (e.g., physical properties) about the foreignobject may be added to an entry for a reference object 130. Future scansof the same patient and/or different patients may then use the newreference object 130 and the data associated with it to successfullyidentify additional instances of the foreign object. Therefore, theaccuracy of the foreign object identifier may continually improve. Ifthe user selects to leave the foreign object (likely indicating that theforeign object is in fact a native object rather than a foreign object),then the intraoral images that include representations of that objectmay be unmodified and the outline or other emphasis of that object inthe view screen may be removed. If the user fails to select the foreignobject, then the original decision to leave the foreign object maystand.

In some instances, foreign objects may be intentionally added to thepatient's mouth for imaging and/or dentistry or orthodontia. In suchinstances, the foreign object identifier 115 may not filter outrepresentations of the foreign object (which may be a reference object).Instead, foreign object identifier 115 may identify the foreign objectand then use known physical properties of the foreign object (e.g., suchas shape, color, etc.) to augment the image with further informationabout the foreign object. Additionally, or alternatively, the knownproperties of the foreign object may be used to augment a virtual 3Dmodel generated from or otherwise associated with the image. Forexample, a user may desire to scan the foreign object to obtain anaccurate and complete representation of the foreign object, but theforeign object may have a geometry that is difficult to fully captureand/or be made of a material that is difficult to image (e.g., such astitanium). Once the foreign object identifier 115 identifies the foreignobject as matching a known reference object 130, foreign objectidentifier 115 may use the known physical properties of the referenceobject to improve the representation of the foreign object in theintraoral images and/or in virtual 3D models. For example, the positionand orientation of the foreign object in the image may be determined,and this position and orientation may be used to add data (e.g., shapedata or surface data) for the foreign object to the image and/or virtual3D model. This process may be useful, for example, for foreign objectssuch as scan bodies, braces, and so on.

Once a foreign object is identified that is to be augmented using areference object 130, information about the foreign object mayadditionally be added to the view screen by user interface 140. Forexample, if a user scans an edge of a scan body and that scan body isthen identified as matching a reference object 130, then unscannedportions of the scan body may be registered with and shown at a correctposition and orientation relative to other features on the dental sitein the view screen. The additional unscanned portions of the scan bodymay be shown in mesh or an outline, for example. This may show a useradditional images to be taken to gather sufficient images to fullycapture the foreign object. As new images are generated, the mesh oroutline representation of the foreign object may be replaced with actualscan data for the foreign object. If ultimately there are any regionsthat a user is unable to image successfully, then the information fromthe reference object 130 may be used to fill in the missing information.Such filled in missing information may be filled in on the intraoralimages and/or on a virtual 3D model generated from or otherwiseassociated with the intraoral images.

FIGS. 1B-9 illustrate methods associated with intraoral scanning as wellas identification and processing of foreign objects in intraoral imagesgenerated during the intraoral scanning. The methods may be performed byprocessing logic that comprises hardware (e.g., circuitry, dedicatedlogic, programmable logic, microcode, etc.), software (such asinstructions run on a processing device), or a combination thereof.Various embodiments may be performed by processing logic executing oncomputing device 105 of FIG. 1A. The methods may be performed inreal-time or near real-time as images are generated by a scanner and/orreceived from the scanner.

FIG. 1B illustrates a flow diagram for a method 152 of identifyingrepresentations of foreign objects in intraoral images, in accordancewith embodiments of the present disclosure. At block 154, regions of apatient's dentition and/or mouth and/or one or more tools may beprepared for intraoral scanning. The preparation of the dentition and/ormouth may include spraying such regions with a dye having a color thatis not natively found inside of a patient's mouth. For example, apatient's tongue may be spray coated, a region of a patient's dentalarch that is not to be scanned may be spray coated, and so on.Similarly, one or more tools that will be inserted into the patient'smouth may be spray coated. Some tools may already be a particular colorthat does not naturally occur in the patient's mouth. For example, adentist may put on gloves having a same color as the dye in the spraycoating.

At block 156, a dental practitioner scans a dental arch (or a portion ofa dental arch) using scanner 150 of FIG. 1A. At block 157, processinglogic processes one or more intraoral images from the intraoral scan toidentify any foreign objects. The intraoral images may be the intraoralimages included in scan data generated by the scanner 150.Alternatively, a 3D model may be generated from intraoral imagesgenerated by the scanner 150. The 3D model may then be projected onto a2D surface to generate a new image that potentially includes data frommultiple intraoral images generated by scanner 150. The new image may beprocessed to identify any foreign objects in the new image. FIGS. 2-9describe various techniques for identifying foreign objects, which maybe used separately or together. At block 158, a user may be presentedwith an option to provide user input with regards to an identifiedforeign object. For example, the user may have the option of marking anidentified foreign object as not being a foreign object, may have theoption of marking an identified native object as being a foreign object,may have the option of confirming an identified foreign object as beinga foreign object, and so on. If the user provides any user input atblock 158, then at block 166 such user input is recorded. Such userinput may then be used to train the foreign object identifier 115, whichis described in greater detail below.

Two different operations may be performed for an identified foreignobject. In a first operation, a virtual object (e.g., a virtual 3D modelor other representation of the foreign object) may be added to one ormore intraoral images and/or associated virtual 3D models at anappropriate location and orientation (block 160). Alternatively, thedetected foreign object may be filtered out (i.e. removed) from theintraoral images (block 162). If there are more than one foreign objectsdetected, then the decision of whether to filter out or snap virtualobjects to the detected foreign objects may be performed separately foreach foreign object.

In an example of snapping a virtual object to an intraoral image and/orvirtual 3D model, a bracket may be identified on a tooth of a patient.The bracket may be difficult to scan, and the image and/or 3D model ofthe bracket may be poor quality. However, the known data for theidentified bracket may be used to complete the representation of thebracket in the image and/or virtual 3D model and to show a perfectbracket (e.g., as it is truly shaped).

At block 164, a virtual model is then generated based on the modifiedintraoral images (e.g., that either include additional data about theforeign object or that do not include any representations of the foreignobject). In some embodiments, rather than snapping virtual objects toimages (at block 160), virtual objects may be instead snapped to avirtual model constructed from one or more intraoral images based on theforeign objects identified in the one or more intraoral images. Forexample, intraoral images used to generate the virtual 3D model may beused to determine the position and orientation of a known object, andthen that information may be used to add information about the knownobject to the virtual 3D model. In another example, an image may begenerated by projecting the virtual 3D model onto a 2D surface, and thatimage may be processed to identify the known object therein and todetermine the position and orientation of the known object. Thedetermined position and orientation of the known object may then be usedto add information about the known object to the virtual 3D model.Alternatively, the virtual object may be snapped both to the intraoralimages and to the virtual 3D model.

As mentioned, at block 180 processing logic may train the foreign objectidentifier to better identify foreign objects in intraoral images. Theforeign object identifier may be trained using data from numerousdifferent sources. In some embodiments, the foreign object identifiercomes pre-trained with default settings 170. Such default settings mayinclude global color data (or other wavelength reflectivity, wavelengthreactivity, wavelength refraction, etc.) for colors that are commonlyfound in the mouth and/or that are not found in the mouth. Such defaultsettings may further include textures, patterns, shapes, etc. that arecommonly found in the mouth and/or that are not found in the mouth. Inone example, a default reference object library may be pre-populatedwith data for a plurality of common dental tools. Each entry in thedefault reference object library may include information such as avirtual 3D model, shape information, color information, surface patterninformation, texture information, etc. about a particular dental tool.

In some embodiments, at block 172 the foreign object identifier mayreceive patient data indicating one or more foreign objects that areincluded in a particular patient's mouth (e.g., specific bracketsattached to the patient's teeth). This may narrow the field of foreignobjects to search for, and search criteria and/or search boundaries maybe adjusted to account for the fact that there is a 100% chance that theindicated foreign object(s) are in the mouth. The foreign objects may beindicated based on a user scanning a bar code, manually entering a codeor product identifier, selecting a foreign object from a drop down menu,and so on. Based on the provided information, the foreign object may beidentified in a reference object library, and physical properties tosearch for intraoral images may be determined.

In some embodiments, at block 174 the foreign object identifier mayreceive an indication of a particular scan procedure that was performed.For example, some types of scan procedures may be to scan for particularinformation. For example, a retainer scan procedure may be performed toscan a patient's mouth to depict a retainer on the patient's dentition.A retainer scan procedure may include retrieving or looking up bracketinformation (e.g., from a reference object library. Similarly, arestorative scan procedure may include retrieving or looking uprestorative object information (e.g., from a reference object library).The foreign object identifier may then adjust its parameters to findbrackets (in the instance of a retainer scan procedure) or to find arestorative object (in the instance of a restorative scan procedure).

In some embodiments, at block 178 a training mode may be entered. Whilethe foreign object identifier is in the training mode, a user may usescanner 150 to generate a 3D scan of a foreign object. This may includegenerating a series of 3D images and/or a 3D video of the object. Theimages and/or video may then be used to generate a virtual 3D model ofthe object. Additionally, one or more image processing algorithms may beapplied to the images and/or video to determine one or more physicalproperties of the object. The images, video, virtual 3D model and/orphysical properties may then be added to a new entry for the foreignobject in a reference object library.

Once the foreign object identifier is trained, it may more accuratelyidentify specific foreign objects at block 157.

FIG. 2 illustrates a flow diagram for a method 200 of filteringrepresentations of foreign objects out of intraoral images, inaccordance with embodiments of the present disclosure. A dentalpractitioner may perform an intraoral scan of a dental site (e.g., ofupper and/or lower dental arches, portions of an upper and/or lowerdental arch, etc.). During the intraoral scan or after the intraoralscan is complete, processing logic may receive scan data comprising oneor more intraoral images of the dental site at block 205 of method 200.Each intraoral image may be a three dimensional (3D) image generated byan intraoral scanner (e.g., scanner 150 of FIG. 1).

At block 210, processing logic analyzes the scan data and identifies arepresentation of a foreign object in the intraoral image based on theanalysis. In some embodiments, analyzing the scan data includesprocessing one or more intraoral images in the scan data using one ormore image processing algorithms. Examples of image processingalgorithms that may be used to process an intraoral image include edgedetection algorithms, 3D object detection algorithms, feature extractionalgorithms (e.g., speeded up robust features (SURF) algorithm,scale-invariant feature transform (SIFT) algorithm, Haar-like featuresalgorithm, histogram of oriented gradients (HOG) algorithm, etc.), andso on. After the intraoral image is processed, shapes, edges, features,and so on may be determined for the intraoral image. The shapes, edges,features, etc. may then be further analyzed to identify native objectsand foreign objects. In an example, foreign objects may often have auniform surface pattern and/or texture on at least a portion of theforeign objects. Such uniform surface pattern and/or texture may be usedto determine a contour of the foreign object, where the contour extendsaround the region with the uniform surface pattern and/or texture.

In one embodiment, at block 215 processing logic analyzes the scan datato determine one or more physical properties represented in theintraoral image. This may include analyzing the intraoral image beforeor after any of the above mentioned image processing algorithms havebeen used on the intraoral image. Multiple different types of physicalproperties may be detected, including reflectivity, diffraction,reactivity to a wavelength of light, texture, surface pattern, colorand/or shape. Other physical properties and/or optical properties mayalso be detected.

In some instances, a foreign object may be identified based solely on asingle physical property (e.g., based on color). For example, dentalinstruments, a dentist's gloves, cotton rolls, etc. may all be dyed acolor that is not naturally occurring in the mouth. For example, thecolor blue does not generally occur naturally in the teeth or gums.Accordingly, a hue of blue may be used. Additionally, a dentalpractitioner may spray objects that will be inserted into the mouth witha dye having a color that is not naturally occurring in the mouth. Thedental practitioner may also spray a patient's tongue with the dye sothat the tongue will be identified as a foreign object based on itscolor. Any region of the intraoral image having the non-natural colorcan easily be identified as belonging to a foreign object with minimalprocessing. A simple color filter may be applied to intraoral images,for example, which filters out one or more colors (e.g., a range ofcolors) that are not naturally occurring in the mouth.

In some instances, a combination of physical properties of an object maybe used to identify that object as a foreign object. For example, theshape of an object and that objects surface pattern or texture may beused together to identify the foreign object. The more physicalproperties that can be identified about an object, the easier it may beto determine whether or not that object is a foreign or native object,and the more accurate that determination may be.

For the reactivity to a wavelength of light data, a specific wavelengthof light may have been shined on a dental site while the intraoral imagewas generated. One example of a wavelength of light that may be used isnear infrared (NIR) light. Other examples include IR light, ultraviolet(UV) light, near ultraviolet (NUV) light, mid infrared (MIR) light, farinfrared (FIR) light, extreme ultraviolet (EUV) light, extreme highfrequency (EHF) electromagnetic radiation, and so on. Native objects ofthe dental site may react to the wavelength of light in one manner, andforeign objects at the dental site may react to the wavelength of lightin a different manner. For example, teeth and tissue in the mouth mayhave particular reflections to NIR light, and metal objects may havevery different reflections to NIR light. For example, foreign objectswould typically appear very dark to NIR light as compared to teeth andtissue. Additionally, blood and saliva may have different reflectancesto NIR light than teeth and tissue, and the use of NIR light may be usedto detect blood and/or saliva in intraoral images. Such blood and/orsaliva may be identified as a foreign object in some embodiments.

In one embodiment, processing logic identifies bubbles in the intraoralimage. Bubbles may be signs of saliva, and generally do not occur aspart of a tooth or gingival tissue. Accordingly, the presence of bubblesmay be another physical property that may be detected and used toidentify saliva and/or blood.

At block 220, processing logic identifies the representation of theforeign object as belonging to a foreign object based on the one or morephysical properties (e.g., based on reflectivity, diffraction,reactivity to a wavelength of light, texture, surface pattern, colorand/or shape). This may include comparing the identified physicalproperties to physical properties of one or more reference objects in areference object library. If a match is found between one or morephysical properties of an object in the intraoral image and one or morephysical properties of a reference object, then the representation ofthe object may be identified as a foreign object.

At block 225, processing logic modifies the intraoral image by removingthe representation of the foreign object from the intraoral image. Atblock 230, processing logic receives additional scan data comprisingadditional intraoral images of the dental site during the intraoralscan. These intraoral images could have been received together with thefirst intraoral image initially before the operations of block 210 wereperformed. Alternatively, one or more additional images may be receivedafter the initial image was received and while (or after) the operationsof block 210 and/or 225 are being performed on the first intraoralimage.

At block 235, processing logic analyzes the additional scan data in thesame manner as described with reference to block 210. At block 240,processing logic then determines whether any of the additional intraoralimages include a representation of the foreign object. If so, then thoseadditional images are also modified at block 245 by removing the foreignobject from the additional intraoral images. Otherwise the methodproceeds to block 255.

At block 255, processing logic registers (i.e., “stitches” together) theintraoral images. This may include registering a first modifiedintraoral image to a second modified intraoral image, or registering afirst modified intraoral image to a second unmodified intraoral image,for example. In one embodiment, performing image registration includescapturing 3D data of various points of a surface in multiple images, andregistering the images by computing transformations between the images.The images may then be integrated into a common reference frame byapplying appropriate transformations to points of each registered image.Processing logic then generates a virtual 3D model of the dental sitebased on the registration using the modified intraoral image and theadditional intraoral images (which may or may not be modified). Thevirtual 3D model may be a digital model showing the surface features ofthe dental site.

FIG. 3 illustrates a flow diagram for a method 300 of identifyingforeign objects in intraoral images, in accordance with embodiments ofthe present disclosure. At block 305 of method 300, processing logicperforms image processing on an intraoral image to determine shapes,textures, surface patterns, colors, reflectivities, diffractions and/orreactivities to wavelengths of light in the intraoral image. The imageprocessing may be performed using edge detection algorithms, featureextraction algorithms and/or other image processing algorithms, such asthose described above.

At block 310, processing logic determines shapes, textures, surfacepatterns, colors, reflectivities, diffractions and/or reactivities towavelengths of light from the intraoral image that are not naturallyoccurring at a dental site. This may include at block 315 comparing theshapes, textures, surface patterns, colors, reflectivities, diffractionsand/or reactivities to wavelengths of light identified in the intraoralimage to shapes, textures, surface patterns, colors, reflectivities,diffractions and/or reactivities to wavelengths of light that are knownto naturally occur in the mouth. At block 320, processing logic may thenidentify shapes, textures, surface patterns, colors, reflectivities,diffractions and/or reactivities to wavelengths of light from theintraoral image that do not naturally occur in the mouth based on thecomparison.

At block 325, processing logic determines a contour of a region of theintraoral image having at least one of the shapes, textures, surfacepatterns, colors, reflectivities, diffractions and/or reactivities towavelengths of light that were determined to not natively occur in themouth. At block 330, processing logic may compare the contour, shapes,textures, surface patterns, colors, reflectivities, diffractions and/orreactivities to wavelengths of light in the intraoral image to knownshapes, textures, surface patterns, colors, reflectivities, diffractionsand/or reactivities to wavelengths of light of reference objects. Atblock 335, processing logic may determine whether the shapes, textures,surface patterns, colors, reflectivities, diffractions and/orreactivities to wavelengths of light in the intraoral image matchshapes, textures, surface patterns, colors, reflectivities, diffractionsand/or reactivities to wavelengths of light of any known referenceobjects in view of the comparison performed at block 330. If a match isfound, the method proceeds to block 340 and the representation of theforeign object is identified. The representation of the foreign objectmay then be removed from the intraoral image. Note that in someembodiments, the shapes, textures, surface patterns, colors,reflectivities, diffractions and/or reactivities to wavelengths of lightthat are not naturally occurring in the mouth may be removed from theintraoral image at block 310 or block 325 without comparing to a libraryof reference objects. If no match is found at block 335, then the methodcontinues to block 345 and no identification of a foreign object ismade.

FIG. 4 illustrates a flow diagram for a method 400 of adding an entryfor a foreign object to a foreign object reference library, inaccordance with embodiments of the present disclosure. At block 405 ofmethod 400, processing logic enters a training mode (e.g., based on userinput). At block 410, processing logic receives one or more images of aforeign object that is to be added to a known reference object library.The images may be received during a 3D scan of the foreign object. Atblock 415, processing logic generates a virtual model of the foreignobject based on the images. Processing logic may also perform imageprocessing on the images and/or the virtual model to determine one ormore physical properties of the foreign object. At block 420, processinglogic adds the virtual model of the foreign object, the images of theforeign object and/or the physical properties of the foreign object to areference library of foreign objects. Thereafter, representations ofthat object will be identified as representations of a foreign object inintraoral images.

FIG. 5 illustrates a flow diagram for a method 500 of using a foreignobject reference library to identify a foreign object in an intraoralimage, in accordance with embodiments of the present disclosure. Atblock 505 of method 500, processing logic receives an indication that aforeign object is at a dental site. The indication may include anidentification of the foreign object, such as by name, uniqueidentifier, characteristics, and so on. For example, an orthodontist mayindicate that a patient is wearing specific brackets for braces. Atblock 510, processing logic queries a reference object library using theidentification of the known object. At block 515, processing logicreceives a response to the query. The response may include one or morephysical properties of the foreign object, a virtual model of theforeign object, metadata associated with an entry of the foreign object,and/or other information. At block 520, processing logic uses the one ormore known properties of the foreign object to identify the foreignobject in the intraoral image. For example, processing logic may searchthe intraoral image for objects having the physical propertiesidentified in the response.

FIG. 6 illustrates a flow diagram for a method 600 of identifyingforeign objects in intraoral images based on color, in accordance withembodiments of the present disclosure. At block 605 of method 600,processing logic selects a pixel (or voxel) in an intraoral image. Atblock 610, processing logic determines, for each color channel of acolor encoding system, at least one of a saturation value or anintensity value associated with the pixel (or voxel). At block 615,processing logic determines a tuple that includes at least one of thesaturation values or the intensity values of each color channel of thecolor encoding system. For example, if the RGB color encoding system isused, then the tuple may be a 3-tuple (r,g,b).

At block 620, processing logic compares the generated tuple to aspecified color range. The specified color range may cover some or allnaturally occurring colors in the mouth. Alternatively, the specifiedcolor range may cover some or all colors not naturally occurring in themouth. At block 625, processing logic determines whether the tuple isoutside of the color range (in the case where the color range indicatescolors naturally occurring in the mouth). If the tuple is outside of thecolor range (in the case where the color range indicates colorsnaturally occurring in the mouth), the method continues to block 630,and the pixel (or voxel) is marked as being potentially associated witha foreign object. If the tuple is inside of the color range, the methodproceeds to block 635.

Alternatively, at block 625 processing logic may determine whether thetuple is inside of the color range (in the case where the color rangeindicates colors not naturally occurring in the mouth). If the tuple isinside of the color range (in the case where the color range indicatescolors not naturally occurring in the mouth), the method continues toblock 630, and the pixel (or voxel) is marked as being potentiallyassociated with a foreign object. If the tuple is outside of the colorrange, the method proceeds to block 635.

At block 635, processing logic determines whether all pixels of theintraoral image have been processed. If there are further pixels toprocess, the method returns to block 605 and another pixel is selectedfor analysis. If all pixels of the intraoral image have been processed,the method continues to block 640. At block 640, processing logicdetermines contiguous pixels or voxels having tuples that are outside ofthe specified color range (in the case where the color range indicatescolors naturally occurring in the mouth) or inside of the specifiedcolor range ((in the case where the color range indicates colors notnaturally occurring in the mouth). At block 645, processing logicdetermines a contour around the determined contiguous pixels or voxels.At block 650, processing logic identifies the pixels or voxels withinthe contour as representations of a portion of a foreign object.Alternatively, the operations of blocks 640-645 may be omitted. Instead,all pixels marked as potentially associated with a foreign object may befiltered out of the intraoral image.

FIG. 7 illustrates a flow diagram for a method 700 of providing a userinterface for management of foreign objects in intraoral images, inaccordance with embodiments of the present disclosure. At block 705 ofmethod 700, processing logic receives scan data comprising an intraoralimage during an intraoral scan of a dental site. At block 710,processing logic identifies a representation of a foreign object in theintraoral image based on an analysis of the scan data. At block 720,processing logic modifies the intraoral image by removing therepresentation of the foreign object from the intraoral image. At block730, during the intraoral scan, processing logic generates a view of thedental side based on the modified image.

At block 735, processing logic generates an outline, meshrepresentation, translucent representation, or other representation ofthe foreign object. At block 745, processing logic adds the outline (orother representation) of the foreign object to the view. At block 750,processing logic provides a user interface that enables a user to selectthe foreign object. At block 755, processing logic receives a userselection of the foreign object. The user selection indicates that theobject should not be removed from the intraoral image (e.g., mayidentify the object as a native object). At block 760, processing logicadds the foreign object back to the intraoral image.

FIG. 8 illustrates a flow diagram for a further method 800 of providinga user interface for management of foreign objects in intraoral images,in accordance with embodiments of the present disclosure. At block 805of method 800, processing logic receives scan data comprising anintraoral image during an intraoral scan of a dental site. At block 810,processing logic identifies a representation of a foreign object in theintraoral image based on an analysis of the scan data. At block 815,processing logic determines a confidence value associated with therepresentation. At block 820, processing logic determines that theconfidence value is below a threshold. If the confidence value is belowthe threshold, this indicates that processing logic is unable toaccurately determine whether or not the identified object should beclassified as a foreign object or a native object.

At block 830, during the intraoral scan, processing logic generates aview of the dental side based on the modified image. At block 835,processing logic generates an outline, mesh representation, translucentrepresentation, or other representation of the possible foreign object.At block 845, processing logic adds the outline (or otherrepresentation) of the foreign object to the view. At block 850,processing logic provides a user interface that enables a user to a)remove the representation of the foreign object from the intraoral imageor b) leave the representation of the object in the intraoral image.

At block 855, processing logic receives a user selection to remove therepresentation from the intraoral image. At block 860, processing logicdetermines one or more physical properties of the foreign object. Atblock 865, processing logic removes the foreign object from theintraoral image. At block 870, processing logic adds an entry for theforeign object to a reference object library. The entry may include thedetermined physical properties of the foreign object. In futureintraoral images the processing logic will then be able to identify theforeign object with increased confidence and accuracy.

In some embodiments, machine learning models such as convolutionalneural networks (CNNs) are used to identify foreign objects in intraoralimages. The intraoral images may be images generated by an intraoralscanner and/or images generated by projecting a virtual 3D model of adental site onto a 2D surface or plane. Either such images may be orinclude height maps. The machine learning models may alternatively, oradditionally, identify foreign objects in virtual 3D models directly(e.g., by inputting data representing the virtual 3D models into themachine learning models). In one example embodiment, a method oftraining a machine learning model includes receiving a training datasetcomprising a plurality of intraoral images, each image of the pluralityof intraoral images comprising a depiction of a dental site and a labelindicating whether or not the intraoral image includes a foreign objecttherein. Intraoral images that include a foreign object may furtherinclude a provided contour of the foreign object. The intraoral imagesmay be real images and/or may be computer generated images. The providedcontours may have been generated using a manual process in which a userhas drawn the contours around foreign objects in the images or anautomated or semi-automated process. In some embodiments, intraoralimages with foreign objects are further labeled with a class, type orcategory of the foreign object. Each type of foreign object may beaccompanied by an appropriate label, for example.

Training the machine learning model includes inputting the trainingdataset into an untrained machine learning model (e.g., CNN or otherneural network) and training the untrained machine learning model basedon the training dataset to generate a trained machine learning modelthat identifies foreign objects in intraoral images. The machinelearning model may also be trained to generate a contour of the foreignobject. In one embodiment, the machine learning model is trained togenerate a binary mask, where a first value of the binary mask indicatespixels that are associated with foreign objects and a second value ofthe binary mask indicates pixels that are associated with nativeobjects. Additionally, for an input image the trained machine learningmodel may output an indication of a type of foreign object that has beenidentified in the intraoral image.

Processing logic processes images from the training dataset one at atime generally. Processing logic processes the intraoral image todetermine an output (a classification or label) and compares thedetermined classification or label (or multiple classifications orlabels) to the provided classification(s) or label(s) associated withthe intraoral image to determine one or more classification error. Anerror term or delta may be determined for each node in the artificialneural network. Based on this error, the artificial neural networkadjusts one or more of its parameters for one or more of its nodes (theweights for one or more inputs of a node). Parameters may be updated ina back propagation manner, such that nodes at a highest layer areupdated first, followed by nodes at a next layer, and so on. Anartificial neural network contains multiple layers of “neurons”, whereeach layer receives as input values from neurons at a previous layer.The parameters for each neuron include weights associated with thevalues that are received from each of the neurons at a previous layer.Accordingly, adjusting the parameters may include adjusting the weightsassigned to each of the inputs for one or more neurons at one or morelayers in the artificial neural network.

In some embodiments, an initial machine learning model is initiallytrained using a small dataset with manually labeled and contouredforeign objects. This machine learning model may then be used toautomatically define contours around foreign objects in additionalimages in a training dataset that will be used to train a more accuratemachine learning model (or to further train the initial machine learningmodel). The contours generated for images in the training dataset may bereviewed by a user, and may be corrected by the user where they areincorrect. The extended dataset may then be used to train a final modelto define contours around foreign objects in intraoral images and toidentify foreign objects in intraoral images. This iterative process ofgenerating the machine learning model that defines contours aroundforeign objects makes it possible to prepare a large training datasetwith minimal user time spent annotating images.

FIG. 9 illustrates a flow diagram for a method 900 of filteringrepresentations of foreign objects out of intraoral images based on useof a trained machine learning model (e.g., a deep learning model,artificial neural network, convolutional neural network, etc.), inaccordance with embodiments of the present disclosure. At block 902 ofmethod 900, processing logic may perform feature extraction on anintraoral image to determine features in the image. This may includeperforming edge detection, object detection, and/or other featureextraction techniques described herein above.

At block 905, processing logic processes the intraoral image (and/or theextracted features of the intraoral image) using a trained machinelearning model. In one embodiment, the trained machine learning model isa deep neural network. In one embodiment, the trained machine learningmodel is a CNN. A convolutional neural network (CNN), for example, hostsmultiple layers of convolutional filters. Pooling is performed, andnon-linearities may be addressed, at lower layers, on top of which amulti-layer perceptron is commonly appended, mapping top layer featuresextracted by the convolutional layers to decisions (e.g. classificationoutputs). Deep learning is a class of machine learning algorithms thatuse a cascade of multiple layers of nonlinear processing units forfeature extraction and transformation. Each successive layer uses theoutput from the previous layer as input. Deep neural networks may learnin a supervised (e.g., classification) and/or unsupervised (e.g.,pattern analysis) manner. Deep neural networks include a hierarchy oflayers, where the different layers learn different levels ofrepresentations that correspond to different levels of abstraction. Indeep learning, each level learns to transform its input data into aslightly more abstract and composite representation. In an imagerecognition application, for example, the raw input may be a matrix ofpixels; the first representational layer may abstract the pixels andencode edges; the second layer may compose and encode arrangements ofedges; the third layer may encode higher level shapes (e.g., teeth,lips, gums, etc.); and the fourth layer may recognize that the imagecontains a face or define a contour around a foreign object in theimage. Notably, a deep learning process can learn which features tooptimally place in which level on its own. The “deep” in “deep learning”refers to the number of layers through which the data is transformed.More precisely, deep learning systems have a substantial creditassignment path (CAP) depth. The CAP is the chain of transformationsfrom input to output. CAPs describe potentially causal connectionsbetween input and output. For a feedforward neural network, the depth ofthe CAPs may be that of the network and may be the number of hiddenlayers plus one. For recurrent neural networks, in which a signal maypropagate through a layer more than once, the CAP depth is potentiallyunlimited. In one embodiment, the machine learning model is aregion-convolution neural network (R-CNN). An R-CNN is a type of CNNthat is able to locate and detect objects in images.

The machine learning model applies a classification or label to theimage based on its current parameter values. An artificial neuralnetwork includes an input layer that consists of values in a data point(e.g., RGB values of pixels in the intraoral image). The next layer iscalled a hidden layer, and nodes at the hidden layer each receive one ormore of the input values. Each node contains parameters (e.g., weights)to apply to the input values. Each node therefore essentially inputs theinput values into a multivariate function (e.g., a non-linearmathematical transformation) to produce an output value. A next layermay be another hidden layer or an output layer. In either case, thenodes at the next layer receive the output values from the nodes at theprevious layer, and each node applies weights to those values and thengenerates its own output value. This may be performed at each layer. Afinal layer is the output layer, where there is one node for each class.For the artificial neural network, there may be a first class (includesforeign object) and a second class (does not include foreign object).Moreover, that class may be determined for each pixel in the image. Foreach pixel in the image, the final layer applies a probability that thepixel of the image belongs to the first class (foreign object) and aprobability that the pixel of the image belongs to the second class(native object). In embodiments where the machine learning model hasbeen trained to identify multiple different types of foreign objects,different classes may be determined for each pixel in the image.

At block 910, processing logic receives an output of the machinelearning model. The output may include a binary mask that has a numberof entries that is equal to a number of pixels or voxels in theintraoral image, wherein entries associated with pixels or voxels thatare part of the foreign object have a first value and wherein entriesassociated with pixels or voxels that are not part of the foreign objecthave a second value. At block 915, processing logic modifies theintraoral image using the mask. Pixels or voxels having the first valuemay be removed from the intraoral image, while pixels or voxels havingthe second value may remain in the intraoral image.

FIG. 10 illustrates a portion of an example dental arch 1005 with aforeign object 1030 disposed thereon. As shown, the portion of thedental arch 1005 includes molar 1010 and molar 1020. Foreign object 1030is a dental mirror, which is disposed at the interproximal regionbetween molar 1010 and molar 1020. Two intraoral images 1058, 1060 havebeen generated of the dental arch 1005.

FIG. 11 illustrates a first intraoral image 1058 that includes a firstrepresentation of the foreign object 1030, in accordance with oneembodiment of the present disclosure. The first intraoral image 1058 maybe processed using one or more of the technique described herein aboveto identify the foreign object 1030 in the intraoral image 1058.

FIG. 12 illustrates a modified version 1068 of the first intraoral imageof FIG. 11, where the foreign object has been filtered out in themodified version 1068 of the first intraoral image, in accordance withone embodiment of the present disclosure. The modified image 1068 maythen be used to generate a virtual 3D model of the dental arch 1005. Byremoving the foreign object 1030 from the intraoral image 1058 prior togenerating the virtual 3D model, the speed of generating the virtualmodel and the accuracy of the virtual model may be improved.

Note that a mirror surface of the dental mirror may include a reflectionof teeth and/or gingival tissue. The reflection may show up as havingphysical properties of native objects in the intraoral image 158.However, the mirror surface is surrounded by a non-reflective border. Inembodiments, processing logic may disregard information that issurrounded by a border having a particular shape and/or meeting othercriteria. This may ensure that reflections of native features that areshown in a reflective foreign object do not confuse processing logicinto incorrectly identifying a foreign object as a native object.

FIG. 13A illustrates view of a portion of an example dental arch 1305with a foreign object 1310 disposed thereon, in accordance with oneembodiment of the present disclosure. The foreign object 1310 may be,for example, a bracket, attachment, scan body, stock implant abutment,or other permanently mounted object. The foreign object may be composedof a material that is difficult to image, such as titanium, in someembodiments.

FIG. 13B illustrates the view of FIG. 13A, where not yet scanned areasof the foreign object 1310 are shown, in accordance with one embodimentof the present disclosure. The foreign object 1310 may have beenidentified using any of the aforementioned techniques. Processing logicmay have then snapped in a full representation of the foreign object tothe view of the dental arch 1305. As shown, the not yet scanned portionsof the foreign object 1310 are shown using a mesh outline. As a usergenerates additional intraoral images of the foreign object, the meshoutline may be filled in with image data from the additional intraoralimages.

Embodiments have been discussed with reference to processing images ofdental sites (e.g., intraoral images) to identify foreign objects (e.g.,reference objects) therein. It should be noted that the same or similartechniques to those described herein above may be used to identifyforeign objects in virtual 3D models generated by stitching togethermultiple intraoral images. Additionally, the images that are processedto identify foreign objects therein may have been generated by anintraoral scanner or may have been generated by projecting a virtual 3Dmodel of a dental site onto a 2D surface or plane. Once a foreign objectis identified, it may be filtered out of an image and/or virtual 3Dmodel or may be augmented in the image and/or virtual 3D model asdescribed above.

FIG. 14 illustrates a diagrammatic representation of a machine in theexample form of a computing device 1400 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. In alternativeembodiments, the machine may be connected (e.g., networked) to othermachines in a Local Area Network (LAN), an intranet, an extranet, or theInternet. The machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a personal computer (PC), a tablet computer, a set-topbox (STB), a Personal Digital Assistant (PDA), a cellular telephone, aweb appliance, a server, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines (e.g., computers)that individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methodologies discussedherein.

The example computing device 1400 includes a processing device 1402, amain memory 1404 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), astatic memory 1406 (e.g., flash memory, static random access memory(SRAM), etc.), and a secondary memory (e.g., a data storage device1428), which communicate with each other via a bus 1408.

Processing device 1402 represents one or more general-purpose processorssuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processing device 1402 may be a complex instructionset computing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 1402may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. Processing device 1402 is configured to execute theprocessing logic (instructions 1426) for performing operations and stepsdiscussed herein.

The computing device 1400 may further include a network interface device1422 for communicating with a network 1464. The computing device 1400also may include a video display unit 1410 (e.g., a liquid crystaldisplay (LCD) or a cathode ray tube (CRT)), an alphanumeric input device1412 (e.g., a keyboard), a cursor control device 1414 (e.g., a mouse),and a signal generation device 1420 (e.g., a speaker).

The data storage device 1428 may include a machine-readable storagemedium (or more specifically a non-transitory computer-readable storagemedium) 1424 on which is stored one or more sets of instructions 1426embodying any one or more of the methodologies or functions describedherein. Wherein a non-transitory storage medium refers to a storagemedium other than a carrier wave. The instructions 1426 may also reside,completely or at least partially, within the main memory 1404 and/orwithin the processing device 1402 during execution thereof by thecomputer device 1400, the main memory 1404 and the processing device1402 also constituting computer-readable storage media.

The computer-readable storage medium 1424 may also be used to store aforeign object identifier 1450, which may correspond to similarly namedforeign object identifier 115 of FIG. 1. The computer readable storagemedium 1424 may also store a software library containing methods thatcall foreign object identifier 1450. While the computer-readable storagemedium 1424 is shown in an example embodiment to be a single medium, theterm “computer-readable storage medium” should be taken to include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore sets of instructions. The term “computer-readable storage medium”shall also be taken to include any medium that is capable of storing orencoding a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresent disclosure. The term “computer-readable storage medium” shallaccordingly be taken to include, but not be limited to, solid-statememories, and optical and magnetic media.

FIG. 15 illustrates a method 1550 of orthodontic treatment using aplurality of appliances, in accordance with embodiments. The method 1550can be practiced using a customized orthodontic appliance or applianceset generated based on a virtual 3D model of a patient's dental arch.The virtual 3D model may have been generated using intraoral images thatwere processed and optionally modified in accordance with embodimentsdescribed herein. In block 1560, a first orthodontic appliance isapplied to a patient's teeth in order to reposition the teeth from afirst tooth arrangement to a second tooth arrangement. In block 1570, asecond orthodontic appliance is applied to the patient's teeth in orderto reposition the teeth from the second tooth arrangement to a thirdtooth arrangement. The method 1550 can be repeated as necessary usingany suitable number and combination of sequential appliances in order toincrementally reposition the patient's teeth from an initial arrangementto a target arrangement. The appliances can be generated all at the samestage or in sets or batches (e.g., at the beginning of a stage of thetreatment), or the appliances can be fabricated one at a time, and thepatient can wear each appliance until the pressure of each appliance onthe teeth can no longer be felt or until the maximum amount of expressedtooth movement for that given stage has been achieved. A plurality ofdifferent appliances (e.g., a set) can be designed and even fabricatedprior to the patient wearing any appliance of the plurality. Afterwearing an appliance for an appropriate period of time, the patient canreplace the current appliance with the next appliance in the seriesuntil no more appliances remain. The appliances are generally notaffixed to the teeth and the patient may place and replace theappliances at any time during the procedure (e.g., patient-removableappliances). The final appliance or several appliances in the series mayhave a geometry or geometries selected to overcorrect the tootharrangement. For instance, one or more appliances may have a geometrythat would (if fully achieved) move individual teeth beyond the tootharrangement that has been selected as the “final.” Such over-correctionmay be desirable in order to offset potential relapse after therepositioning method has been terminated (e.g., permit movement ofindividual teeth back toward their pre-corrected positions).Over-correction may also be beneficial to speed the rate of correction(e.g., an appliance with a geometry that is positioned beyond a desiredintermediate or final position may shift the individual teeth toward theposition at a greater rate). In such cases, the use of an appliance canbe terminated before the teeth reach the positions defined by theappliance. Furthermore, over-correction may be deliberately applied inorder to compensate for any inaccuracies or limitations of theappliance.

FIG. 16 illustrates a method 1600 for designing an orthodonticappliance. Some or all of the blocks of the method 1600 can be performedby any suitable data processing system or device, e.g., one or moreprocessors configured with suitable instructions. In block 1610, amovement path to move one or more teeth from an initial arrangement to atarget arrangement is determined. The initial arrangement can bedetermined from a mold or a scan of the patient's teeth or mouth tissue,e.g., using wax bites, direct contact scanning, x-ray imaging,tomographic imaging, sonographic imaging, and other techniques forobtaining information about the position and structure of the teeth,jaws, gums and other orthodontically relevant tissue. From the obtaineddata, a digital data set can be derived that represents the initial(e.g., pretreatment) arrangement of the patient's teeth and othertissues. Optionally, the initial digital data set is processed tosegment the tissue constituents from each other. For example, datastructures that digitally represent individual tooth crowns can beproduced. Advantageously, digital models of entire teeth can beproduced, including measured or extrapolated hidden surfaces and rootstructures, as well as surrounding bone and soft tissue.

The target arrangement of the teeth (e.g., a desired and intended endresult of orthodontic treatment) can be received from a clinician in theform of a prescription, can be calculated from basic orthodonticprinciples, and/or can be extrapolated computationally from a clinicalprescription. With a specification of the desired final positions of theteeth and a digital representation of the teeth themselves, the finalposition and surface geometry of each tooth can be specified to form acomplete model of the tooth arrangement at the desired end of treatment.

Having both an initial position and a target position for each tooth, amovement path can be defined for the motion of each tooth. In someembodiments, the movement paths are configured to move the teeth in thequickest fashion with the least amount of round-tripping to bring theteeth from their initial positions to their desired target positions.The tooth paths can optionally be segmented, and the segments can becalculated so that each tooth's motion within a segment stays withinthreshold limits of linear and rotational translation. In this way, theend points of each path segment can constitute a clinically viablerepositioning, and the aggregate of segment end points can constitute aclinically viable sequence of tooth positions, so that moving from onepoint to the next in the sequence does not result in a collision ofteeth.

In block 1620, a force system to produce movement of the one or moreteeth along the movement path is determined. A force system can includeone or more forces and/or one or more torques. Different force systemscan result in different types of tooth movement, such as tipping,translation, rotation, extrusion, intrusion, root movement, etc.Biomechanical principles, modeling techniques, forcecalculation/measurement techniques, and the like, including knowledgeand approaches commonly used in orthodontia, may be used to determinethe appropriate force system to be applied to the tooth to accomplishthe tooth movement. In determining the force system to be applied,sources may be considered including literature, force systems determinedby experimentation or virtual modeling, computer-based modeling,clinical experience, minimization of unwanted forces, etc.

The determination of the force system can include constraints on theallowable forces, such as allowable directions and magnitudes, as wellas desired motions to be brought about by the applied forces. Forexample, in fabricating palatal expanders, different movement strategiesmay be desired for different patients. For example, the amount of forceneeded to separate the palate can depend on the age of the patient, asvery young patients may not have a fully-formed suture. Thus, injuvenile patients and others without fully-closed palatal sutures,palatal expansion can be accomplished with lower force magnitudes.Slower palatal movement can also aid in growing bone to fill theexpanding suture. For other patients, a more rapid expansion may bedesired, which can be achieved by applying larger forces. Theserequirements can be incorporated as needed to choose the structure andmaterials of appliances; for example, by choosing palatal expanderscapable of applying large forces for rupturing the palatal suture and/orcausing rapid expansion of the palate. Subsequent appliance stages canbe designed to apply different amounts of force, such as first applyinga large force to break the suture, and then applying smaller forces tokeep the suture separated or gradually expand the palate and/or arch.

The determination of the force system can also include modeling of thefacial structure of the patient, such as the skeletal structure of thejaw and palate. Scan data of the palate and arch, such as X-ray data or3D optical scanning data, for example, can be used to determineparameters of the skeletal and muscular system of the patient's mouth,so as to determine forces sufficient to provide a desired expansion ofthe palate and/or arch. In some embodiments, the thickness and/ordensity of the mid-palatal suture may be measured, or input by atreating professional. In other embodiments, the treating professionalcan select an appropriate treatment based on physiologicalcharacteristics of the patient. For example, the properties of thepalate may also be estimated based on factors such as the patient'sage—for example, young juvenile patients will typically require lowerforces to expand the suture than older patients, as the suture has notyet fully formed.

In block 1630, an orthodontic appliance configured to produce the forcesystem is determined. Determination of the orthodontic appliance,appliance geometry, material composition, and/or properties can beperformed using a treatment or force application simulation environment.A simulation environment can include, e.g., computer modeling systems,biomechanical systems or apparatus, and the like.

In block 1640, instructions for fabrication of the orthodontic applianceor a mold that will be used to manufacture the orthodontic appliance aregenerated. The instructions can be configured to control a fabricationsystem or device in order to produce the orthodontic appliance and/orthe mold. In some embodiments, the instructions are configured formanufacturing the orthodontic appliance using direct fabrication (e.g.,stereolithography, selective laser sintering, fused deposition modeling,3D printing, continuous direct fabrication, multi-material directfabrication, etc.), in accordance with the various methods presentedherein. In alternative embodiments, the instructions can be configuredfor indirect fabrication of the appliance, e.g., by direct 3D printingof the mold, and then thermoforming a plastic sheet over the mold.

Method 1600 may comprise additional blocks: 1) The upper arch and palateof the patient is scanned intraorally to generate three dimensional dataof the palate and upper arch; 2) The three dimensional shape profile ofthe appliance is determined to provide a gap and teeth engagementstructures as described herein.

FIG. 17 illustrates a method 1700 for digitally planning an orthodontictreatment and/or design or fabrication of an appliance, in accordancewith embodiments. The method 1700 can be applied to any of the treatmentprocedures described herein and can be performed by any suitable dataprocessing system.

In block 1710, a digital representation of a patient's teeth isreceived. The digital representation can include surface topography datafor the patient's intraoral cavity (including teeth, gingival tissues,etc.). The surface topography data can be generated by directly scanningthe intraoral cavity, a physical model (positive or negative) of theintraoral cavity, or an impression of the intraoral cavity, using asuitable scanning device (e.g., a handheld scanner, desktop scanner,etc.).

In block 1720, one or more treatment stages are generated based on thedigital representation of the teeth. The treatment stages can beincremental repositioning stages of an orthodontic treatment proceduredesigned to move one or more of the patient's teeth from an initialtooth arrangement to a target arrangement. For example, the treatmentstages can be generated by determining the initial tooth arrangementindicated by the digital representation, determining a target tootharrangement, and determining movement paths of one or more teeth in theinitial arrangement necessary to achieve the target tooth arrangement.The movement path can be optimized based on minimizing the totaldistance moved, preventing collisions between teeth, avoiding toothmovements that are more difficult to achieve, or any other suitablecriteria.

In block 1730, at least one orthodontic appliance is fabricated based onthe generated treatment stages. For example, a set of appliances can befabricated, each shaped according a tooth arrangement specified by oneof the treatment stages, such that the appliances can be sequentiallyworn by the patient to incrementally reposition the teeth from theinitial arrangement to the target arrangement. The appliance set mayinclude one or more of the orthodontic appliances described herein. Thefabrication of the appliance may involve creating a digital model of theappliance to be used as input to a computer-controlled fabricationsystem. The appliance can be formed using direct fabrication methods,indirect fabrication methods, or combinations thereof, as desired.

In some instances, staging of various arrangements or treatment stagesmay not be necessary for design and/or fabrication of an appliance. Asillustrated by the dashed line in FIG. 17, design and/or fabrication ofan orthodontic appliance, and perhaps a particular orthodontictreatment, may include use of a representation of the patient's teeth(e.g., receive a digital representation of the patient's teeth 1710),followed by design and/or fabrication of an orthodontic appliance basedon a representation of the patient's teeth in the arrangementrepresented by the received representation.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other embodiments will beapparent upon reading and understanding the above description. Althoughembodiments of the present disclosure have been described with referenceto specific example embodiments, it will be recognized that thedisclosure is not limited to the embodiments described, but can bepracticed with modification and alteration within the spirit and scopeof the appended claims. Accordingly, the specification and drawings areto be regarded in an illustrative sense rather than a restrictive sense.The scope of the disclosure should, therefore, be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A method comprising: receiving intraoral scandata comprising a plurality of images of a dental site; generating avirtual three-dimensional (3D) model of the dental site based on theplurality of images; performing an analysis on an image of the dentalsite; identifying a representation of a reference object in the image,wherein the reference object has one or more known properties; andmodifying at least one of the image or the virtual 3D model of thedental site by adding additional data about the reference object to atleast one of the image or the virtual 3D model based on the one or moreknown properties of the reference object.
 2. The method of claim 1,wherein the image is one of the plurality of images of the dental siteused to generate the virtual 3D model, wherein the image is modified byadding the additional data about the reference object to the image, andwherein the image, as modified, is used to generate the virtual 3D modelof the dental site.
 3. The method of claim 2, further comprising:performing image processing on the image to determine a probable objectrepresented in the image, wherein the probable object comprises one ormore physical properties; comparing the one or more physical propertiesof the probable object to the one or more known properties of thereference object; determining that the one or more physical propertiesof the probable object match the one or more known properties thereference object as a result of the comparing; and determining that theprobable object is the reference object.
 4. The method of claim 3,wherein the one or more physical properties comprise at least one of areflectivity, a diffraction, a reactivity to a wavelength of light, atexture, a surface pattern, a color or a shape.
 5. The method of claim1, wherein performing the analysis of the image comprises inputting theimage into a machine learning model trained to identify one or moretypes of reference objects, wherein the machine learning model outputsan indication of the representation of the reference object in theimage.
 6. The method of claim 1, further comprising: generating theimage on which the analysis is performed by projecting the virtual 3Dmodel onto a plane.
 7. The method of claim 6, further comprising: duringan intraoral scan, generating a view of the dental site based on themodified image, wherein the view comprises an outline of the referenceobject based on the additional data about the reference object;receiving a plurality of additional images of the dental site during theintraoral scan; replacing one or more parts of a shape of the referenceobject based on the plurality of additional images; and updating theview of the dental site, the updated view showing those parts of theshape of the reference object that have been replaced.
 8. The method ofclaim 1, wherein identifying the representation of the reference objectcomprises: processing the image using a machine learning model that hasbeen trained to identify one or more foreign objects at dental sites;and receiving an output of the machine learning model, wherein theoutput comprises a binary mask that has a number of entries that isequal to a number of pixels or voxels in the image, wherein entriesassociated with pixels or voxels that are part of the reference objecthave a first value and wherein entries associated with pixels or voxelsthat are not part of the reference object have a second value.
 9. Themethod of claim 1, further comprising performing the following beforereceiving the intraoral scan data: entering a training mode; receiving aplurality of images of the reference object during the training mode;generating a virtual model of the reference object based on theplurality of images of the reference object; and adding at least one ofthe virtual model of the reference object or the plurality of images ofthe reference object to a reference object library that comprisesentries for a plurality of reference objects.
 10. The method of claim 1,further comprising: receiving an indication that the reference object isat the dental site, the indication comprising an identification of thereference object; querying a reference object library comprising entriesfor a plurality of reference objects using the identification of thereference object; receiving a response to the query, the responsecomprising data associated with the reference object; and using the datato identify the reference object in the image.
 11. The method of claim1, wherein the one or more known properties comprise at least one of aknown reflectivity of the reference object, a known diffraction of thereference object, a known reactivity of the reference object to awavelength of light, a known texture of the reference object, a knownsurface pattern of the reference object, a known color of the referenceobject or a known shape of the reference object
 12. The method of claim1, further comprising: determining a confidence value associated withthe reference object; determining that the confidence value is below aconfidence threshold; presenting an option to a) add the additional dataabout the reference object to at least one of the image or the virtual3D model or b) leave a representation of the reference object in atleast one of the image or the virtual 3D model unchanged; and receivinga user selection to add the additional data about the reference objectto at least one of the image or the virtual 3D model.
 13. The method ofclaim 1, wherein the reference object comprises at least one of a shapeor a material that causes intraoral scans of the reference object tohave a reduced accuracy.
 14. A non-transitory computer readable mediumcomprising instructions that, when executed by a processing device,cause the processing device to perform operations comprising: receivingintraoral scan data comprising an intraoral image of a dental site;performing an analysis of the intraoral image to a) identify of aforeign object represented in the intraoral image and b) determine thatthe foreign object is an instance of a reference object having one ormore known properties; modifying the intraoral image by addingadditional data about the foreign object to the intraoral image based onthe one or more known properties of the reference object; receivingadditional intraoral scan data comprising a plurality of additionalintraoral images of the dental site; and generating a virtualthree-dimensional (3D) model of the dental site based on the modifiedintraoral image and the plurality of additional intraoral images. 15.The non-transitory computer readable medium of claim 14, whereinperforming the analysis of the intraoral image comprises: performingimage processing on the intraoral image to determine a plurality ofprobable objects in the intraoral image, wherein each of the pluralityof probable objects comprises one or more physical properties;comparing, for each of the plurality of probable objects, the one ormore physical properties of the probable object to a plurality of knownproperties of reference objects; determining that one or more physicalproperties of a probable object of the plurality of probable objectsmatch the one or more known properties the reference object as a resultof the comparing; and determining that the probable object is theforeign object at the dental site.
 16. The non-transitory computerreadable medium of claim 15, wherein the one or more physical propertiescomprise at least one of a reflectivity, a diffraction, a reactivity toa wavelength of light, a texture, a surface pattern, a color or a shape.17. The non-transitory computer readable medium of claim 14, wherein theintraoral scan data and the additional intraoral scan data is receivedduring an intraoral scan, wherein the additional data comprises a shapeof the foreign object, and wherein at least a portion of the shape wasnot shown in the intraoral image, the operations further comprising:during the intraoral scan, generating a view of the dental site based onthe modified intraoral image, wherein the view comprises an outline ofthe foreign object based on the additional data about the foreignobject; replacing one or more parts of the shape of the foreign objectbased on the plurality of additional intraoral images; and updating theview of the dental site, the updated view showing those parts of theshape of the foreign object that have been replaced.
 18. Thenon-transitory computer readable medium of claim 17, wherein at least aportion of the foreign object represented in the virtual 3D model isbased on the additional data about the foreign object and is not basedon a received intraoral image.
 19. The non-transitory computer readablemedium of claim 14, the operations further comprising performing thefollowing before receiving the intraoral scan data: entering a trainingmode; receiving a plurality of images of the foreign object during thetraining mode; generating a virtual model of the foreign object based onthe plurality of images; and adding at least one of the virtual model ofthe foreign object or the plurality of images of the foreign object to areference object library that comprises entries for a plurality ofreference objects.
 20. The non-transitory computer readable medium ofclaim 14, the operations further comprising: receiving an indicationthat the foreign object is at the dental site, the indication comprisingan identification of the foreign object; querying a reference objectlibrary comprising entries for a plurality of reference objects usingthe identification of the foreign object; receiving a response to thequery, the response comprising one or more known properties of theforeign object, the one or more known properties comprising at least oneof a known reflectivity of the foreign object, a known diffraction ofthe foreign object, a known reactivity of the foreign object to awavelength of light, a known texture of the foreign object, a knownsurface pattern of the foreign object, a known color of the foreignobject or a known shape of the foreign object; and using the one or moreknown properties of the foreign object to identify the foreign object inthe intraoral image.
 21. The non-transitory computer readable medium ofclaim 14, the operations further comprising performing the followingprior to modifying the intraoral image: determining a confidence valueassociated with the foreign object; determining that the confidencevalue is below a confidence threshold; presenting an option to a) addthe additional data about the foreign object to the intraoral image orb) leave a representation of the foreign object in the intraoral imageunchanged; and receiving a user selection to add the additional dataabout the foreign object to the intraoral image.
 22. The non-transitorycomputer readable medium of claim 21, the operations further comprising:adding an entry for the foreign object to a reference object library ofknown foreign objects.
 23. A system comprising: a handheld scanner toperform an intraoral scan; and a non-transitory computer readable mediumcomprising instructions that, when executed by a processor, cause theprocess to perform operations comprising: receiving an intraoral imageassociated with an intraoral scan of a dental site; identifying arepresentation of a foreign object in the intraoral image based on atleast one of a reflectivity of the foreign object, a diffraction of theforeign object, a reactivity of the foreign object to a wavelength oflight, a texture of the foreign object, a surface pattern of the foreignobject, a color of the foreign object or a shape of the foreign object;modifying the intraoral image by adding a more complete representationof the foreign object to the intraoral image; receiving a plurality ofadditional intraoral images of the dental site during the intraoralscan; and generating a virtual three-dimensional (3D) model of thedental site using the modified intraoral image and the plurality ofadditional intraoral images.